[Doc Generation] Support parsing `posonlyargs` when get function signature #7578

algorithm1832 · 2025-10-24T16:37:27Z

PR types

others

PR changes

Docs

Description

Add parsing for posonlyargs in api document generation script
Update default value assignment for posonlyargs and args

issue #7435

【详细报告】
在调查8号文档产生问题的过程中，发现英文文档生成过程报错，报错后仍然继续进行build
以下内容摘自Docs-NEW流水线的日志

2025-10-24 15:30:11 Traceback (most recent call last):
2025-10-24 15:30:11   File "/FluidDoc/docs/api/./gen_doc.py", line 1082, in <module>
2025-10-24 15:30:11     set_source_code_attrs()
2025-10-24 15:30:11   File "/FluidDoc/docs/api/./gen_doc.py", line 207, in set_source_code_attrs
2025-10-24 15:30:11     parse_module_file(mod)
2025-10-24 15:30:11   File "/FluidDoc/docs/api/./gen_doc.py", line 322, in parse_module_file
2025-10-24 15:30:11     gen_functions_args_str(node)
2025-10-24 15:30:11   File "/FluidDoc/docs/api/./gen_doc.py", line 382, in gen_functions_args_str
2025-10-24 15:30:11     str_args_list[defarg_ind_start + defarg_ind] += "=" + defarg_val
2025-10-24 15:30:11 IndexError: list index out of range

排查发现，产生报错的文件为Paddle仓库的python/paddle/library.py文件
错误为函数解析错误，产生错误的函数是custom_op和register_fake
以下为register_fake函数的声明

def register_fake(
    op: str | CustomOpDef,
    func: Callable[..., object] | None = None,
    /,
    *,
    lib: None = None,
    _stacklevel: int = 1,
    allow_override: bool = False,
)

该函数只有posonlyargs和kwonlyargs，没有普通的args，而gen_doc.py中并没有对posonlyargs进行处理
在分配参数默认值defaults时，实际str_args_list为空，目标位置为-1，因此报错

本次修改新增了posonlyargs的处理逻辑

if node.args.posonlyargs is not None:
    for arg in node.args.posonlyargs:
        if not arg.arg == "self":
            str_args_list.append(arg.arg)
    # posonlyargs与普通args的分界符是斜杠
    str_args_list.append("/")

更新了分配参数默认值的逻辑

defarg_ind_start = len(str_args_list) - len(node.args.defaults)
# 新增部分1
# 如果默认值数量超出args数量，则说明存在posonlyargs的默认值，为了绕过斜杠需要向前移动一位
if node.args.posonlyargs is not None and len(node.args.defaults) > len(
    node.args.args
):
    defarg_ind_start -= 1
# 新增部分1结束

for defarg_ind in range(len(node.args.defaults)):
    # 新增部分2
    # 如果当前位置是斜杠，说明posonlyargs默认值已经全部分配完毕，为了绕过斜杠需要向后移动一位
    if str_args_list[defarg_ind_start + defarg_ind] == "/":
        defarg_ind_start += 1
    # 新增部分2结束
    if isinstance(node.args.defaults[defarg_ind], ast.Name):
        str_args_list[defarg_ind_start + defarg_ind] += "=" + str(
            node.args.defaults[defarg_ind].id
        )
    elif isinstance(node.args.defaults[defarg_ind], ast.Constant):
        defarg_val = str(node.args.defaults[defarg_ind].value)
        if isinstance(node.args.defaults[defarg_ind].value, str):
            defarg_val = f"'{defarg_val}'"
        str_args_list[defarg_ind_start + defarg_ind] += "=" + defarg_val

已确认该报错在Docs-NEW运行中不再出现，然而此问题并非导致paddle.static.Program英文文档渲染缺少内容的原因，后续会继续调查

paddle-bot · 2025-10-24T16:37:33Z

感谢你贡献飞桨文档，文档预览构建中，Docs-New 跑完后即可预览，预览链接：http://preview-pr-7578.paddle-docs-preview.paddlepaddle.org.cn/documentation/docs/zh/api/index_cn.html
预览工具的更多说明，请参考：飞桨文档预览工具

Copilot

Pull Request Overview

This PR fixes a bug in the API documentation generation script that caused an IndexError when parsing Python functions with position-only arguments (posonlyargs). The issue occurred when processing functions like register_fake in paddle/library.py that only had posonlyargs and kwonlyargs without regular args.

Key Changes:

Added parsing logic for position-only arguments (posonlyargs)
Updated default value assignment logic to handle defaults for posonlyargs
Added logic to skip the "/" separator when assigning default values

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-27T03:30:00Z

docs/api/gen_doc.py

+
        for defarg_ind in range(len(node.args.defaults)):
+            if str_args_list[defarg_ind_start + defarg_ind] == "/":
+                defarg_ind_start += 1


This check is inside the loop and modifies defarg_ind_start each iteration, which can cause incorrect indexing for subsequent defaults. The slash skip should only occur once when first encountered. Consider moving this check outside the loop or using a flag to ensure it only executes once.

Suggested change

for defarg_ind in range(len(node.args.defaults)):

if str_args_list[defarg_ind_start + defarg_ind] == "/":

defarg_ind_start += 1

# Skip the "/" marker only once if it is at the start index

if (

defarg_ind_start < len(str_args_list)

and str_args_list[defarg_ind_start] == "/"

):

defarg_ind_start += 1

for defarg_ind in range(len(node.args.defaults)):

Copilot · 2025-10-27T03:30:00Z

docs/api/gen_doc.py

+            defarg_ind_start -= 1
+
        for defarg_ind in range(len(node.args.defaults)):
+            if str_args_list[defarg_ind_start + defarg_ind] == "/":


Potential IndexError if defarg_ind_start + defarg_ind exceeds the list bounds. This can occur when there are more defaults than combined posonlyargs and args. Add a bounds check before accessing the list element.

Copilot · 2025-10-27T03:30:00Z

docs/api/gen_doc.py


        defarg_ind_start = len(str_args_list) - len(node.args.defaults)
+        if node.args.posonlyargs is not None and len(node.args.defaults) > len(
+            node.args.args
+        ):


The calculation of defarg_ind_start adjustment is unclear. The logic assumes defaults overflow into posonlyargs territory by exactly one position, but this may not account for all edge cases. Consider adding a comment explaining the mathematical relationship between defaults, args, posonlyargs, and the '/' separator position.

Suggested change

defarg_ind_start = len(str_args_list) - len(node.args.defaults)

if node.args.posonlyargs is not None and len(node.args.defaults) > len(

node.args.args

):

# Calculate the starting index in str_args_list where default values should be applied.

# The list str_args_list contains argument names in the order: posonlyargs (if any), '/', args.

# node.args.defaults contains default values for the last N positional arguments (posonlyargs + args).

# The defaults are applied to the last N arguments, which may include both args and posonlyargs.

# If the number of defaults exceeds the number of args, the extra defaults apply to posonlyargs.

# The '/' separator is present if posonlyargs exist, and must be skipped when assigning defaults.

defarg_ind_start = len(str_args_list) - len(node.args.defaults)

if node.args.posonlyargs is not None and len(node.args.defaults) > len(

node.args.args

):

# If defaults overflow into posonlyargs, adjust the start index to skip the '/' separator.

sunzhongkai588

LGTM
@SigureMo 一师傅要不要看看，提点建议

sunzhongkai588 · 2025-10-27T06:22:33Z

已确认该报错在Docs-NEW运行中不再出现，然而此问题并非导致paddle.static.Program英文文档渲染缺少内容的原因，后续会继续调查

paddle.static.Program 文档用的是老的 IR （python\paddle\base\framework.py），已经被新的 IR 替换，但文档还没有更新，你可以尝试更新一下～ @algorithm1832

SigureMo · 2025-10-27T06:50:37Z

docs/api/gen_doc.py

                            logger.debug("%s omitted", obj_full_name)


 def gen_functions_args_str(node):


def gen_function_args_string(fn_node: ast.FunctionDef) -> str: def _gen_default_value_string(default_node: ast.expr) -> str: if isinstance(default_node, ast.Constant): default_value = default_node.value if isinstance(default_value, str): return f"'{default_value}'" return str(default_value) elif isinstance(default_node, ast.Name): return default_node.id return "..." def _is_self_arg(arg: ast.arg) -> bool: return arg.arg == "self" if not isinstance(fn_node, ast.FunctionDef): warnings.warn(f"The provided node {ast.unparse(fn_node)} is not a FunctionDef.") return "" arg_list = [] posonlyargs_count = len(fn_node.args.posonlyargs) for i, arg in enumerate(fn_node.args.posonlyargs): if _is_self_arg(arg): continue arg_list.append(arg.arg) if posonlyargs_count > 0: arg_list.append("/") for i, arg in enumerate(fn_node.args.args): if _is_self_arg(arg): continue arg_list.append(arg.arg) defaults_start_index = len(fn_node.args.args) - len(fn_node.args.defaults) for i, default in enumerate(fn_node.args.defaults): arg_index = defaults_start_index + i arg_name = fn_node.args.args[arg_index].arg default_value_str = _gen_default_value_string(default) arg_list[arg_index] = f"{arg_name}={default_value_str}" if fn_node.args.vararg: arg_list.append(f"*{fn_node.args.vararg.arg}") if fn_node.args.kwonlyargs: arg_list.append("*") for i, kwonlyarg in enumerate(fn_node.args.kwonlyargs): kwdefault = fn_node.args.kw_defaults[i] assert kwdefault is not None default_value_str = _gen_default_value_string(kwdefault) arg_list.append(f"{kwonlyarg.arg}={default_value_str}") if fn_node.args.kwarg: arg_list.append(f"**{fn_node.args.kwarg.arg}") return ", ".join(arg_list)

直接重写一下吧

defaults_start_index = len(fn_node.args.args) - len(fn_node.args.defaults)

这个好像不对吧，当时那个解析出错的函数没有args，但存在posonlyargs，然后默认值是分配给posonlyargs的
所以defaults_start_index不但需要考虑args，还需要考虑posonlyargs和后面那个斜杠参数

最新的commit已针对此问题进行调整

defaults_start_index = ( len(fn_node.args.args) + len(fn_node.args.posonlyargs) - len(fn_node.args.defaults) ) for i, arg in enumerate(fn_node.args.posonlyargs + fn_node.args.args): if _is_self_arg(arg) and i == 0: continue if i == posonlyargs_count and posonlyargs_count > 0: arg_list.append("/") if i >= defaults_start_index: default_value_str = _gen_default_value_str( fn_node.args.defaults[i - defaults_start_index] ) arg_list.append(f"{arg.arg}={default_value_str}") else: arg_list.append(arg.arg)

那就这么写

这个也改好了

algorithm1832 · 2025-10-27T08:01:08Z

Docs-NEW报了一个AssertionError，发现是python\paddle\profiler\profiler.py中的make_scheduler函数

def make_scheduler(
    *,
    closed: int,
    ready: int,
    record: int,
    repeat: int = 0,
    skip_first: int = 0,
) -> Callable[[int], ProfilerState]

并不是所有的kwonlyargs都分配了默认值，至于为什么原先的代码能正确识别默认值，是因为这个函数的kw_defaults长这样：

[None, None, None, <ast.Constant object at 0x711b5545f450>, <ast.Constant object at 0x711b5545f350>]

SigureMo · 2025-10-27T08:40:53Z

docs/api/gen_doc.py

    for i, kwonlyarg in enumerate(fn_node.args.kwonlyargs):
        kwdefault = fn_node.args.kw_defaults[i]
-        assert kwdefault is not None
        default_value_str = _gen_default_value_string(kwdefault)


不是直接 remove assert 就行的

if kwdefault is None: arg_list.append(kwonlyarg.arg) else: default_value_str = _gen_default_value_string(kwdefault) arg_list.append(f"{kwonlyarg.arg}={default_value_str}")

改好了，前面看太急了，不好意思喵

SigureMo

LGTMeow

add parsing for posonlyargs

330b364

paddle-bot bot added the contributor label Oct 24, 2025

algorithm1832 changed the title ~~[WIP] [Doc Generation] Add parsing for posonlyargs~~ [Doc Generation] Add parsing for posonlyargs Oct 25, 2025

algorithm1832 marked this pull request as ready for review October 25, 2025 03:23

luotao1 added the HappyOpenSource 快乐开源活动issue与PR label Oct 27, 2025

luotao1 assigned luotao1, sunzhongkai588 and ooooo-create Oct 27, 2025

sunzhongkai588 requested a review from Copilot October 27, 2025 03:29

Copilot AI reviewed Oct 27, 2025

View reviewed changes

sunzhongkai588 approved these changes Oct 27, 2025

View reviewed changes

SigureMo reviewed Oct 27, 2025

View reviewed changes

algorithm1832 added 2 commits October 27, 2025 15:20

merge branch develop

26363bc

refactor code

fabf85a

algorithm1832 added 3 commits October 27, 2025 16:23

remove assert

b154e08

add None default value processing

db898fd

fix argument name processing

fbb6fd9

SigureMo reviewed Oct 27, 2025

View reviewed changes

algorithm1832 added 2 commits October 27, 2025 16:47

add None default value processing

9ca407f

refactor code

76f7408

SigureMo changed the title ~~[Doc Generation] Add parsing for posonlyargs~~ [Doc Generation] Support parsing posonlyargs when get function signature Oct 27, 2025

SigureMo approved these changes Oct 27, 2025

View reviewed changes

SigureMo merged commit a9812f5 into PaddlePaddle:develop Oct 27, 2025
4 checks passed

-        defarg_ind_start = len(str_args_list) - len(node.args.defaults)
-        if node.args.posonlyargs is not None and len(node.args.defaults) > len(
-            node.args.args
-        ):
+        # Calculate the starting index in str_args_list where default values should be applied.
+        # The list str_args_list contains argument names in the order: posonlyargs (if any), '/', args.
+        # node.args.defaults contains default values for the last N positional arguments (posonlyargs + args).
+        # The defaults are applied to the last N arguments, which may include both args and posonlyargs.
+        # If the number of defaults exceeds the number of args, the extra defaults apply to posonlyargs.
+        # The '/' separator is present if posonlyargs exist, and must be skipped when assigning defaults.
+        defarg_ind_start = len(str_args_list) - len(node.args.defaults)
+        if node.args.posonlyargs is not None and len(node.args.defaults) > len(
+            node.args.args
+        ):
+            # If defaults overflow into posonlyargs, adjust the start index to skip the '/' separator.

		logger.debug("%s omitted", obj_full_name)


		def gen_functions_args_str(node):

[Doc Generation] Support parsing posonlyargs when get function signature #7578

[Doc Generation] Support parsing posonlyargs when get function signature #7578

Uh oh!

Conversation

algorithm1832 commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR types

PR changes

Description

Uh oh!

paddle-bot bot commented Oct 24, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

sunzhongkai588 left a comment

Choose a reason for hiding this comment

Uh oh!

sunzhongkai588 commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SigureMo Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

algorithm1832 Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

SigureMo Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

algorithm1832 Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

algorithm1832 commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SigureMo Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

algorithm1832 Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

SigureMo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[Doc Generation] Support parsing `posonlyargs` when get function signature #7578

[Doc Generation] Support parsing `posonlyargs` when get function signature #7578

algorithm1832 commented Oct 24, 2025 •

edited

Loading

sunzhongkai588 commented Oct 27, 2025 •

edited

Loading

algorithm1832 commented Oct 27, 2025 •

edited

Loading